Using historical information to improve search across heterogeneous indices

ABSTRACT

A method, system and computer program product are disclosed for searching for data. In one embodiment, the invention provides a method comprising identifying a query and a search scope including a set of specified entities; and for each of these entities, estimating a number of documents that would be identified in a search through the entity to answer the query. On the basis of this estimating, a subset of the entities is formed. The query and this subset of entities are sent to a search engine to search the subset of entities to answer the query. In one embodiment, the estimating includes collecting statistical information from queries to build up a historical cache using heuristics or machine learning techniques, wherein the query includes a key word and a scope, and the historical cache contains a maximum number of returned results for an entity given the queries executed.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of copending application Ser. No.14/672,860 filed Mar. 30, 2015, which is a continuation of U.S. patentapplication Ser. No. 12/535,330, filed Aug. 4, 2009, now U.S. Pat. No.8,996,561 issued Mar. 31, 2015, the entire content and disclosure ofwhich is hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention generally relates to data processing, and morespecifically, to searching for data or information in order to answer aquery. Even more specifically, embodiments of the invention relate tomethods, apparatus and computer program products that are well suitedfor retrieving information across heterogeneous indices.

Description of the Related Art

The Internet and the World Wide Web have become critical, integral partsof commercial operations, personal lives, and the education process. Atthe heart of the Internet is web browser technology and Internet servertechnology. An Internet server contains “content” such as documents,image or graphics files, forms, audio clips, etc., all of which isavailable to systems and browsers which have Internet connectivity. Webbrowser or “client” computers may request documents from web addresses,to which appropriate web servers respond by transmitting one or more webdocuments, image or graphics files, forms, audio clips, etc. The mostcommon protocol for transmission of web documents and contents fromservers to browsers is Hyper Text Transmission Protocol (“HTTP”).

The most common type of Internet content or document is Hyper TextMarkup Language (“HTML”) documents, but other formats are also wellknown in the art, such as Adobe Portable Document Format (“PDF”). HTML,PDF and other web documents provide “hyperlinks” within the document,which allow a user to select another document or web site to view.Hyperlinks are specially marked text or areas in the document which whenselected by the user, command the browser software to retrieve or fetchthe indicated document or to access a new web site. Ordinarily, when theuser selects a plain hyperlink, the current page being displayed in theweb browser's graphical user interface (“GUI”) window disappears and thenewly received page is displayed. If the parent page is an index, forexample the IBM web site www.patents.ibm.com, and the user wishes tovisit each descending link (e.g. read the document with tips on how touse the site), then the parent or index page disappears and the new pageis displayed (such as the help page).

As the computing capacity of web browser computers increases and thecommunications bandwidth to the web browser computer increasesdramatically, one challenge for organizations that provide Internet websites and content is to deliver and filter such content in anticipationof these greater processing and throughput speeds. This is particularlytrue in the realm of web-based applications, and in the development ofbetter and more efficient ways to move user-pertinent information to thedesktop or client. However, today's web browsers are in generalunintelligent software packages. As these browsers currently exist, theyrequire the user to manually search for any articles or documents ofinterest to him or her, and these browsers are often cumbersome in thatthey frequently require a download of many documents before one ofgermane interest is found.

Search engines provide some level of “intelligence” to the browsingexperience, wherein a user may point his unintelligent web browser to asearch engine address, enter some keywords for a search, and then revieweach of the returned documents one at a time by selecting hyperlinks inthe search results, or by re-pointing the web browser manually toprovided web addresses. However, search engines do not really search theentire Internet, rather they search their own indices of Internetcontent which has been built by the search engine operator, usuallythrough a process of reviewing manual submissions from other web siteoperators. Thus, it is common for a user to use several search engineswhile looking for information on a particular subject, because eachsearch engine will return different results based on its own indexcontent.

To address this problem, another technology has been developed and isknown in the art as “MetaSearch engine”. A MetaSearch engine does notkeep its own index, but rather submits a query to multiple searchengines simultaneously, and returns to the user the highest rankedreturns from each of these search engines. The MetaSearch engine may,though, return the top 5 listings from 4 search engines, which mayfilter out the more likely interesting information.

MetaSearch engines are constructed to support unified access to multiplesearch engines. With reference to FIG. 1, when merging results frommultiple search indices 20, 22, a MetaSearch engine 24 can adopt eitherlocal similarity adjustment or global similarity estimation to providedocuments 26. In the local adjustment approach, each component searchengine ranks documents locally. Then the MetaSearch engine normalizesthe ranks into the same range with additional information such as thequality of component search engines. For global similarity estimation,the MetaSearch engine computes a global similarity score for eachreturned document with certain information from component engines, suchas the local document frequency of a term. Today a number of MetaSearchengines have been constructed and are available on the internet such asMetaCrawler and Dogpile. The component search engines in these systemsdeal with the same type of data, the document level indices. Documentsin these systems as shown in FIG. 1 are first class entities. The term“first class entities” refers to the entities that can be used inprograms without restrictions. Here, it refers to the abstract objects(such as books and departments) used in the system designed.

IBM's Enterprise Information Leverage (EIL) system can be regarded as aMetaSearch engine which provides unified access to services engagementdata. A service engagement represents the interaction as well as thedocuments exchanged between sellers and clients. With reference to FIG.2, an EIL system 30 combines information extraction and semantic searchto support information needs of a user. An EIL system leveragesstructured and unstructured data using novel architecture and specialpurpose algorithms. Information is organized around an entity (such asengagements, books and departments), and the system supports a semanticconcept index based information retrieval 32 by utilizing bothinformation of first class entities in database queries 34 and ofdocument index search 36 where the relevant entities act as a contextualconstraint 38. In EIL systems, there is a need to deal withheterogeneous search indices; these indices are associated withdocuments as well as semantic concepts extracted from these documents.These concepts represent important properties of a service engagement.Analogously, a system can include data about books and each page in abook, or about departments in a company and each person in thedepartment, etc. Furthermore, the indices can be stored in differentplaces. For example, data about books can be stored in relationaldatabases such as DB2, and information about pages in the books can bestored in a search engine such as OmniFind. The semantical differencesbetween heterogeneous search indices may be a problem when merging andranking the results in a MetaSearch engine.

Furthermore, in systems similar to the EIL system, documents are notfirst class entities. These entities can be engagements, books,departments, and so on. For instance, a user may want to search for abook about Java programming. If a page of content in a book mentionsJava programming, the book should be returned. The ideal result is thata number of books are returned that relate to Java programming where,under each book, the top ranked pages containing the keywords are listedwith hit highlights. Based on the hit highlights and the properties ofbooks, a user can decide if a book is of interest. Therefore, it isimportant to cover as many books as possible given a certain number ofbook pages.

For example, two search indices for 5000 books have been established.One index is a keyword search index that is stored in a keyword searchengine. The other index has specific properties of each book, such asthe book titles, authors' names, dates published, abstracts, readers'comments, and so on. Normally, only a limited number of documents can beretrieved from a keyword search engine. For example, by default,OmniFind returns 500 document links for each search call. However, for asearch of the term “Java programming”, a return of 500 pages from thesame book is not the best result. An ideal result would be to have about10 to 20 pages returned for a single book to allow the system to rankthe books based on both the pages that are returned and the properties(semantic concepts) indexed in a relational database. In this way, thereare a sufficient number of books presented for the user withoutretrieving too many pages. In a regular web search engine, documents arestored as first class entities and there is no need to group documentsinto a higher level of entities. What is needed is a system and searchengine processing methodology that presents a sufficient number of booksto a user for review without retrieving an excessively large number ofpages.

SUMMARY OF THE INVENTION

This invention is directed to a system and method for improving therecall of search results and minimizing search cost withoutsignificantly affecting the precision of the search, while consideringseveral constraints (for example, the limitation of query length incertain search engines). Embodiments of the invention provide a method,system and computer program product for searching for data. In oneembodiment, the invention provides a method comprising identifying aquery and a search scope including a set of specified entities; and foreach of said specified entities, estimating a number of documents thatwould be identified in a search through said each entity to answer saidquery. On the basis of this estimating, a subset of the entities isformed, and the query and this subset of entities are sent to a searchengine to search said subset of entities to answer said query.

In another embodiment, the invention provides a system for searching fordata. This system comprises one or more processing units configured forreceiving a query and a search scope including a set of specifiedentities; and for each of said specified entities, estimating a numberof documents that would be identified in a search through said eachentity to answer said query. On the basis of this estimating, a subsetof the entities is formed, and the query and this subset of entities aresent to a search engine to search said subset of entities to answer saidquery.

In another embodiment, the invention provides a computer programproduct, readable by a computer, and, when executed on the computer, thecomputer program product receives a query and a search scope including aset of specified entities; and for each of said specified entities,estimates a number of documents that would be identified in a searchthrough said each entity to answer said query. On the basis of thisestimating, a subset of the entities is formed, and the query and thissubset of entities are sent to a search engine to search said subset ofentities to answer said query.

In one embodiment, the estimating includes collecting statisticalinformation from queries to build up a historical cache using heuristicsor machine learning techniques; wherein said query includes a key wordand a scope, and said historical cache contains a maximum number ofreturned results for an entity given the queries executed. In thisembodiment, the forming includes rewriting said query based on thehistorical cache; and the search engine executes the query to get agroup of entities, each having a group of documents, and the historicalcache is updated with the rewritten query results. Also, for example,the subset of entities may be formed so that the total of the estimatednumber of documents for all of the entities in the subset is not morethan a given number.

BRIEF DESCRIPTION OF THE DRAWINGS

Other aspects, features, and advantages of the present invention willbecome more fully apparent from the following detailed description, theappended claims, and the accompanying drawings in which similar elementsare given similar reference numerals.

FIG. 1 is a block diagram of a prior art meta search engine;

FIG. 2 is a block diagram to illustrate the EIL search system;

FIG. 3 is a block diagram showing functional details of a systemembodying this invention;

FIG. 4 is a flow chart of a method for performing a MetaSearch inaccordance with the principles of the invention;

FIG. 5 shows an adaptive MetaSearch algorithm that may be used in thepresent invention;

FIG. 6 is a flow chart of a method for performing PickEntities inaccordance with the principles of the invention;

FIG. 7 shows an algorithm that may be used to implement the methodoutlined in FIG. 6; and

FIG. 8 is a block diagram of a computer system for use with the presentinvention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 3 shows a block diagram of a search engine system includingfunctional elements and interactions within a web server 40 inaccordance with one embodiment of the present invention. An agentmanager 42 running on server 40 receives domain-specific queries from auser, typically from an input from device 44 such as a mobile device.The user chooses, in one embodiment, one of a number of historicalcaches 46 that are available on server 40 (or which are imported to theserver from other sources), depending on the particular domain of thequery. Additionally or alternatively, the user may identify sites or Webpages on the sites that contain information relevant to a query,typically by inputting sample Uniform Resource Locators (URLs) to anagent.

Mobile device 44 sends user query to the agent manager 42, the managersends the query to the agent 50, and then the agent sends refinedqueries back to the manager 42. Subsequently, the manager 42 sends therefined queries to the search engines 54, the search engines return theresults back to the manager, and then the manager sends the searchengine results to the agent 50. The agent correlates the results, sortsthem at the entity level and sends the results back to the manager 42.Then the manager sends the sorted results back to the mobile device 44.

The knowledge base 56 contains the historical cache, which has thestatistical information collected from query results and/or componentsearch engines. The knowledge base 56 may also contain the domainspecific vocabulary, which is a repository of terms that appear in thehigh-ranking sites of the domain. Each term is preferably associatedwith a list of lexical affinities, other closely related terms that arefrequently found in proximity to that term. Methods for finding lexicalaffinities in a corpus of documents are known in the art. For example,for any given word in a sentence, all other words that are within thesentence and no more than five words away from the given word can beconsidered as its lexical affinities.

For each domain, knowledge base 56 can have the form of a file or set offiles. Thus, to import or export any knowledge base from one server 40to another, and/or from one user to another, it is sufficient to copythe appropriate knowledge base files. Thereafter, the user receiving theknowledge base can personalize the associated knowledge agent bycarrying out further focused searches in his or her specific domain. Asthe user performs more and more such searches, the knowledge agent willbecome increasingly specialized in the particular domain of interest tothe user.

This invention is directed toward minimizing the search cost andimproving the recall of search results without significantly affectingthe precision of the search, while considering several constraints whichare typical in a MetaSearch system. For example, one constraint might bethat each component search engine has specific query limitations. Forexample, with OmniFind, one of the component search engines in thesystem cannot accept queries that contain too many terms. In addition,it is limited to return at most 500 document links for each search call.

A second constraint can be that the number of calls to each componentsearch engine should be reduced to minimize the cost of a search. Athird constraint can relate to privacy and security concerns. Typically,in an enterprise search engine, a user is authorized to have access toonly certain kinds of data based his or her job roles. For example, asecurity policy may indicate that users can only access the documents inthose services engagements that they have worked on.

Specifically, in the IBM EIL system, where security policy is an issue,each user may have access to a portion of the engagement data that isdefined as the search scope. The goal is to return as many engagements(entities) as possible in the scope while minimizing the number of callsto component search engines. In addition, all of the returnedengagements (entities) should be relevant to the query because somedocuments in the engagements contain the query terms. For example, foreach engagement (entity) “d” in the scope, the agent rewrites the searchquery to use “d” as a new scope, and the query is then sent to acomponent search engine such as, for example, OmniFind. This methodguarantees coverage for each engagement in the scope. However, sendingthe query to a component search engine for each engagement will resultin a slow run time for the search.

With a different approach, multiple engagements (entities) are randomlygrouped together as new scopes, and the user or agent re-writes thesearch query for each of the new scopes prior to sending the queries tocomponent search engines. This approach will reduce the number of callsto component search engines, but it cannot guarantee coverage of all ofthe engagements (entities) that are to be searched. This is because someengagements may return a large number of document links for the querywhere the document links occupy the limited slots for the returned linksfrom a component engine such as OmniFind.

Using the book example discussed above, suppose a user is looking forbooks about “Java programming” which were published in 2000. Including“2000” in the keyword search will not help because a book may includethe term “2000” in the content, which is not its published date.Therefore, the term “2000” is searched in the database containing bookproperties, and the result is combined with the returned document links.Using four books as an example, it is assumed that books 1 and 3 werepublished in 1999, book 2 was published in 2000, and book 4 waspublished in 2003. Suppose OmniFind is used as a component search engineand returns only 500 document links for a search call. Furthermore,assume that book 1 has 300 pages that are to be returned, book 2 has 50pages that are to be returned, book 3 has 200 pages that are to bereturned, and book 4 has 100 pages that are to be returned. Normally,OmniFind will return documents in the order of relevance. If books 1, 2and 3 are grouped together as a first scope, and book 4 as a secondscope, it is likely that the query for the first scope only will returndocuments from books 1 and 3 as the 500 page limit will be reached.Therefore, book 2 which may be a good match, will be missed in theresults. If, however, books 2, 3 and 4 are grouped together because itis known that the total number of returned documents, the number ofpages, is less than 500, then book 2 will not be removed from theresults.

We use the above example to illustrate the steps in the algorithmsAdaptiveMetaSearch and PickEntities as follows. FIG. 4 is a flow chart100 of a MetaSearch in accordance with the principles of the invention(the algorithm AdaptiveMetaSearch). The flow chart includes symbols anddata structures defined as follows:

Cache H represents a data structure for recording the collectedstatistical information. Here we use it to record the maximum number ofreturned documents for an entity, such as an engagement or a book, givenall of the user queries submitted,

i.e., H(d)=MAX(H(d), q(d)),

where d is an entity, such as an engagement or a book; q(d) is thereturned number of documents of d from the results of the most recentquery q; MAX(para1, para2) is a function that compares para1 and para2,and returns the bigger one as the result; H(d) represents the maximumnumber of returned documents for d collected so far and its initialvalue could be zero. The cache H can also be constructed by using otherheuristics or machine learning techniques. In the book example, thecache H gives the estimation of how many pages each book might returngiven a query. The cache H can be used to determine which entitiesshould be grouped together and sent to a component search engine beforethe other entities in a search scope. In addition, it is assumed thatthe cache is always ranked in ascending order.

Threshold T₁: the total number of returned documents for entities in agroup should be no more than T₁, and T₁ shall have a value that is noless than the maximum number of documents a component search enginereturns for a query. In the book example, T₁ can be set to 500.Threshold T₂: if the number of different entities between the set ofentities to be covered by a query and the set of returned entities ofthe query is smaller than T₂, then there is no need to get the next setof document links from the search engine. In the book example, this canbe set to 1. The entities that have not been covered by the searchresults can be combined with entities in the next scope and sent to thesearch engine as a new query.

The input of algorithm AdaptiveMetaSearch includes a query Q 101, whichcomprises terms to be searched, such as “Java Programming”; D 102, a setof entities as a scope, such as book 1, book 2, book 3 and book 4; H103, the cache, which has a cache value representing an estimated numberof documents for an entity with respect to Q, and two thresholds T₁ andT₂. The output is a list of returned entities, and within each entity, alist of obtained ranked documents.

FIG. 4 is a flow chart of the algorithm AdaptiveMetaSearch, and FIG. 5shows pseudo-code for this algorithm. As shown in FIGS. 4 and 5, thealgorithm AdaptiveMetaSearch first calls the function “PickEntities,” asdiscussed below, to select a subset L of entities from the scope D, step104. Then the algorithm initializes M to be an empty set, step 106, andsends the query Q and entity set L to a component search engine, step108. The first set of document links returned by the search engine isdesignated as N, step 110, and at step 112, a check is made to determineif the set N is empty. If N is not empty, the routine proceeds to step114, where entities from the document links in N are added to M. At thistime, at step 116, the cardinality of the difference between L and M iscompared to T₂. If this cardinality of difference is not less than orequal to T₂, then at step 120, N becomes the next set of document linksreturned by the search engine. At step 122, N is compared to M todetermine if the entities from N are already in M. If not, the processreturns to step 114; however, if the entities from N are already in M,then, at step 124, M is subtracted from D (notice that the entities in Lbut not in M are still in D and will be picked up in subsequent scopes).Also, if at step 116 the cardinality of difference between L and M isless than or equal to T₂, then the routine proceeds from step 116 tostep 124, where M is subtracted from D.

After step 124, a check is made at step 126 to determine if the set D isempty. If D is empty, then at step 128, the entities in M and theirranked document links are returned. If at step 126, D is not empty, thenthe routine moves on to step 130, and another call to PickEntitiesmethod is performed to select a new subset L of entities from D. Fromstep 130, the routine returns to step 108. The algorithm of FIG. 4 alsoproceeds to step 126 from step 112 if, at step 112, N is empty. Inparticular, if at step 112, N is empty, then L is subtracted from D atstep 132, and the routine then goes to step 126.

In the book example, the function PickEntities first returns book 2,book 4 and book 3 as a sub-scope. Then this sub-scope together with thequery are sent to the component search engine. Then based on thereturned documents, book 2, book 3 and book 4 are added into M. Then thescope D is updated such that only book 1 is left. The second call ofPickEntities returns book 1 as the sub-scope and book 1 is added into M.Then D becomes empty and the search process stops. Eventually M containsthe four books and the corresponding document links (returned pages).

FIG. 6 shows a flow chart 200 that illustrates the PickEntities methodin accordance with the principles of the invention and FIG. 7 showspseudo-code for the algorithm. In this embodiment, the method usesinputs D, a set of entities (the scope) 202, cache H 204, and thresholdT₁ 206. For example, given the four books discussed above and theestimation from the cache H, the function PickEntities may return book2, book 4 and book 3 as a sub-scope. At step 208, let C be the list ofentities in H, in ascending order of the estimated number of returneddocuments. Then C is updated to comprise the intersection between C andD, step 210, so that only the entities in the scope D are considered inlater steps. In addition, the variable L is set to be an empty set, step212, and d is set to be the first entity in C, step 214. For each entityd, a determination is made at step 216 as to whether the relationshipH(d)+Σ_(d′∈L) H(d′)≤T₁ is satisfied, i.e., if the number (H(d)) ofreturned documents of d plus the total number (Σ_(d′∈L) H(d′)) ofreturned documents of the existing entities in set L is equal to or lessthan T₁.

If, at step 216, this sum is less than T₁, then d is added to set L,step 218. At step 220, it is determined if there is any new entity in Cwhich has not been considered. If the answer is YES, d becomes the nextentity in C, step 222, and the process goes back to step 216. Ifhowever, it is determined, at step 220, that there is no new entity inC, then it is determined, at step 224, if L is empty and C is not empty.If the answer is no, then L is returned, step 226. If, however, theanswer is yes at step 224, then the first entity of C is added to L,step 228. Returning to step 216, if the relationship in step 216 is notsatisfied, then the process proceeds to step 224 where it is determinedif L is empty and C is not empty.

In the above-discussed book example, suppose the scope D is books 1, 2,3 and 4; and suppose H estimates book 1 has 300 pages to be returned,book 2 has 50 pages, book 3 has 200 pages and book 4 has 100 pages. ThenC has book 2, then book 4, then book 3, then book 1 in the ascendingorder of the estimated returned numbers. The set L acquires book 2, book4 and book 3 with a total number 350 of returned pages (documents).Then, when book 1 comes, the check in step 216 will fail, because 350plus 300 equals 650, which is larger than 500, the threshold T₁.Therefore, the first call of PickEntities returns book 2, book 4 andbook 3 as a sub-scope.

The present invention can be used on any properly configured generalpurpose computer system, such as the system shown in FIG. 8. Such acomputer system 300 includes a processing unit (CPU) 302 connected by abus 301 to a random access memory 304, a high density storage device308, a keyboard 306, a display 310 and a mouse 312. In addition, thereis a floppy disk drive 314 and a CD-ROM drive 316 for entry of data andsoftware, including software embodying the present invention, into thesystem on removable storage. An example of such a computer is an IBMPersonal computer of the International Business Machines Corporation,such as an Aptiva personal computer operating on Microsoft Windowsoperating system of the Microsoft Corporation. Also in this examplethere is an internet browser capable at running Java such as NetscapeNavigator of the Netscape Communications Corporation or InternetExplorer of the Microsoft Corporation.

The various method embodiments of the invention will be generallyimplemented by a computer executing a sequence of program instructionsfor carrying out the steps of the method, assuming all required data forprocessing is accessible to the computer. The sequence of programinstructions may be embodied in a computer program product comprisingmedia storing the program instructions. As will be readily apparent tothose skilled in the art, the present invention can be realized inhardware, software, or a combination of hardware and software. Any kindof computer/server system(s)—or other apparatus adapted for carrying outthe methods described herein—is suited. A typical combination ofhardware and software could be a general-purpose computer system with acomputer program that, when loaded and executed, carries out the method,and variations on the method as described herein. Alternatively, aspecific computer, containing specialized hardware for carrying out oneor more of the functional tasks of the invention, could be utilized.

As will be appreciated by one skilled in the art, the present inventionmay be embodied as a system, method or computer program product.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer-usableprogram code embodied in the medium.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM) or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, ofotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave, The computer usable program code may betransmitted using any appropriate medium, including but not limited towireless, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, though the Internet using an Internet Service Provider).

The present invention is described above with reference to flow chartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flow chart illustrationsand/or block diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Although an example of the present invention has been shown anddescribed, it would be appreciated by those skilled in the art thatchanges might be made in the embodiment without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

What is claimed is:
 1. A method of searching for data, comprising:identifying a query and a search scope including a set of entities, eachof the entities including one or more documents; forming a subset ofsaid entities based on defined criteria; and sending said query and saidsubset of said entities to a search engine to search said subset of saidentities; the search engine searching through said subset of saidentities to identify a first group of documents from said subset of saidentities, said first group of documents linking to a number of theentities in said subset of the entities; and when the number of thelinked entities is less than a defined value, the search enginesearching through said subset of said entities to identify a secondgroup of documents from said subset of said entities.
 2. The methodaccording to claim 1, further comprising: when said number of the linkedentities is more than the defined value, forming another subset of theentities and sending said query and said another subset of the entitiesto the search engine to search said another subset of said entities forfurther documents.
 3. The method according to claim 2, furthercomprising: for each of said specified entities, estimating a number ofthe documents included in said each entity that would be identified in asearch through said each entity to answer said query; and the forming asubset of said entities includes forming the subset of said entitiesbased on the estimated number of the documents included in said eachentity that would be identified in a search through said each entity toanswer said query.
 4. The method according to claim 3, wherein: saidestimating a number of the documents includes storing in a historicalcache results from previous searches through each of the entities,including for each of the entities, storing in the historical cache anumber that is the largest number of the documents identified in theeach entity during any of the previous searches through the each entity.5. The method according to claim 4, wherein the forming a subset of saidincludes forming the subset of said entities based on said numbersstored in the historical cache.
 6. The method according to claim 5,wherein the forming the subset of said entities based on said numbersstored in the historical cache includes using said number, from thehistorical cache, that is the largest number of the documents identifiedin the each entity during the any of the previous searches through theeach entity, as an estimated number of return documents for included insaid each entity.
 7. The method according to claim 1, wherein the totalof the estimated number of documents for all of the entities in thesubset of entities is not more than a given number.
 8. The methodaccording to claim 7, wherein the forming the subset of said entitiesfurther includes: arranging the entities in a defined order; and formingthe subset of the entities based on said defined order.
 9. The methodaccording to claim 8, wherein: the arranging includes arranging theentities in order based on said number, from the historical cache, thatis the largest number of the documents identified in each of thespecified entities during any of the previous searches through the eachentity; and the forming includes adding a defined number of the entitiesto the subset of entities, in ascending order of said number, from thehistorical cache, that is the largest number of the documents identifiedin each of the specified entities during any of the previous searchesthrough the each entity.
 10. The method according to claim 9, wherein:the total of the estimated number of documents that would be returnedfor the entries in the subset is less than a given number; theidentifying a query includes using a mobile device to transmit thequery, and using a server to receive the query, and using a server toreceive the query from the mobile device; the sending the query includesoperating an agent manager on the server to send the query to the searchengine; and the method further comprises: the search engine returningresults of the searching to the agent manager; the agent manager sendingthe results to an agent operating on the server; using the agent tocorrelate the results, sort the results and send the sorted results backto the agent manager; and the agent manager sending the sorted resultsto the mobile device.
 11. A system for searching for data, comprisingone or more processing units configured for: receiving a query and asearch scope including a set of entities, each of the entities includingone or more documents; forming a subset of said entities based ondefined criteria; and sending said query and said subset of saidentities to a search engine to search said subset of said entities; thesearch engine searching through said subset of said entities to identifya first group of documents from said subset of said entities, said firstgroup of documents linking to a number of the entities in said subset ofthe entities; and when the number of the linked entities is less than adefined value, the search engine searching through said subset of saidentities to identify a second group of documents from said subset ofsaid entities.
 12. The system according to claim 11, wherein said one ormore processing units are further configured for: when said number ofthe linked entities is more than the defined value, forming anothersubset of the entities and sending said query and said another subset ofthe entities to the search engine to search said another subset of saidentities for further documents.
 13. The system according to claim 12,wherein said one or more processing units are further configured for:for each of said specified entities, estimating a number of thedocuments included in said each entity that would be identified in asearch through said each entity to answer said query; and the forming asubset of said entities includes forming the subset of said entitiesbased on the estimated number of the documents included in said eachentity that would be identified in a search through said each entity toanswer said query.
 14. The method according to claim 13, wherein: saidestimating a number of the documents includes storing in a historicalcache results from previous searches through each of the entities,including for each of the entities, storing in the historical cache anumber that is the largest number of the documents identified in theeach entity during any of the previous searches through the each entity.15. The method according to claim 14, wherein the forming a subset ofsaid includes forming the subset of said entities based on said numbersstored in the historical cache.
 16. An article of manufacturecomprising: at least one computer usable device having computer readableprogram code logic tangibly embodied therein to execute instructions ina processing unit for searching for data, said computer readable programcode logic, when executing, performing the following: receiving a queryand a search scope including a set of entities, each of the entitiesincluding one or more documents; forming a subset of said entities basedon defined criteria; and sending said query and said subset of saidentities to a search engine to search said subset of said entities; thesearch engine searching through said subset of said entities to identifya first group of documents from said subset of said entities, said firstgroup of documents linking to a number of the entities in said subset ofthe entities; and when the number of the linked entities is less than adefined value, the search engine searching through said subset of saidentities to identify a second group of documents from said subset ofsaid entities.
 17. The article of manufacture according to claim 16,said computer readable program code logic, when executing, furtherperforming: when said number of the linked entities is more than thedefined value, forming another subset of the entities and sending saidquery and said another subset of the entities to the search engine tosearch said another subset of said entities for further documents. 18.The article of manufacture according to claim 17, said computer readableprogram code logic, when executing, further performing: for each of saidspecified entities, estimating a number of the documents included insaid each entity that would be identified in a search through said eachentity to answer said query; and the forming a subset of said entitiesincludes forming the subset of said entities based on the estimatednumber of the documents included in said each entity that would beidentified in a search through said each entity to answer said query.19. The article of manufacture according to claim 18, wherein: saidestimating a number of the documents includes storing in a historicalcache results from previous searches through each of the entities,including for each of the entities, storing in the historical cache anumber that is the largest number of the documents identified in theeach entity during any of the previous searches through the each entity.20. The article of manufacture according to claim 19, wherein theforming a subset of said includes forming the subset of said entitiesbased on said numbers stored in the historical cache.